Evaluating Web Search Result Summaries
نویسندگان
چکیده
The aim of our research is to produce and assess short summaries to aid users’ relevance judgements, for example for a search engine result page. In this paper we present our new metric for measuring summary quality based on representativeness and judgeability, and compare the summary quality of our system to that of Google. We discuss the basis for constructing our evaluation methodology in contrast to previous relevant open evaluations, arguing that the elements which make up an evaluation methodology: the tasks, data and metrics, are interdependent and the way in which they are combined is critical to the effectiveness of the methodology. The paper discusses the relationship between these three factors as implemented in our own work, as well as in SUMMAC/MUC/DUC.
منابع مشابه
Evaluating the Effectiveness of Visual Summaries for Web Search
With ever-increasing amounts of information on the World Wide Web, an effective interface for displaying search results is required. Recent studies have developed various novel approaches for visual summaries, aiming to improve the effectiveness of search results. In this study we evaluate the effectiveness of four types of visual summary: thumbnails, salient images, visual snippets and visual ...
متن کاملAn Analysis of User Strategies for Examining and Processing Ranked Lists of Documents
The predominant display of document retrieval results is a ranked list of query-biased summaries. When examining and processing search results, users must make complex decisions about how to allocate their time and relevance judging effort between evaluation of summaries and the full documents reachable with a mouse click on a summary. We performed a cluster analysis of the search results proce...
متن کاملEnhanced Visualization for Web-Based Summaries
For each search result presented by a search engine, a user has a choice to click through for more information or to skip the result. We aim to improve the accuracy of this click process by introducing a color-coding scheme built upon our improved summary text selection approach called ReClose. Color-coding adds an additional level of context to the text without requiring additional screen spac...
متن کاملImproving Search Results Quality by Customizing Summary Lengths
Web search engines today typically show results as a list of titles and short snippets that summarize how the retrieved documents are related to the query. However, recent research suggests that longer summaries can be preferable for certain types of queries. This paper presents empirical evidence that judges can predict appropriate search result summary lengths, and that perceptions of search ...
متن کاملCan Automatic Abstracting Improve on Current Extracting Techniques in Aiding Users to Judge the Relevance of Pages in Search Engine Results?
Current search engines use sentence extraction techniques to produce snippet result summaries, which users may find less than ideal for determining the relevance of pages. Unlike extracting, abstracting programs analyse the context of documents and rewrite them into informative summaries. Our project aims to produce abstracting summaries which are coherent and easy to read thereby lessening use...
متن کامل